feat: backend quota and auto-compaction spec fields, NOSPACE alarm surfacing#401
feat: backend quota and auto-compaction spec fields, NOSPACE alarm surfacing#401xrl wants to merge 6 commits into
Conversation
spec.quotaBackendBytes, spec.autoCompactionMode and spec.autoCompactionRetention render the corresponding etcd flags. Flags are emitted only when set so existing clusters keep byte-identical args; etcdOptions still wins on conflict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Lange <xrlange@gmail.com>
Poll etcd alarms after the member health check. An active NOSPACE alarm sets Degraded reason DatabaseQuotaExceeded (overriding the generic UnhealthyMembers) and emits a Warning event per member. Detection only: no defrag or disarm automation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Lange <xrlange@gmail.com>
mgr.GetEventRecorder emits events.k8s.io/v1 Events; the core-group events rule alone leaves the DatabaseQuotaExceeded event forbidden. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Lange <xrlange@gmail.com>
Steady-state reconciles never patch the pod template, so the NOSPACE event and docs must not imply raising spec.quotaBackendBytes is applied immediately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Lange <xrlange@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: xrl The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @xrl. Thanks for your PR. I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
- quotaBackendBytes below 100Mi (CEL quantity floor): etcd disables the
quota for non-positive values, and a byte-scale typo ("8" for "8Gi")
raises a cluster-wide NOSPACE on the first pod restart.
- a non-integer autoCompactionRetention in revision mode: etcd falls
back to time.ParseDuration and uses the raw nanosecond count as the
revision retention, so compaction effectively never runs.
Regenerated CRD, chart CRD and API docs.
Signed-off-by: Xavier Lange <xrlange@gmail.com>
(cherry picked from commit 785e8d7)
…rollout' The operator re-renders the template on any spec edit and starts a quorum-gated one-pod-at-a-time restart at once; under an active NOSPACE alarm the health gates hold it until the alarm is disarmed. Fix the doc and event text, and add the operator-downgrade NOSPACE caveat. Signed-off-by: Xavier Lange <xrlange@gmail.com> (cherry picked from commit eefd979)
What
Database-size lifecycle basics for
EtcdCluster, in two halves:Spec knobs for quota and compaction. Three new optional fields —
spec.quotaBackendBytes(resource.Quantity),spec.autoCompactionMode(enumperiodic/revision),spec.autoCompactionRetention(pattern-validated duration or revision count) — render the matching--quota-backend-bytes/--auto-compaction-mode/--auto-compaction-retentioncontainer args. Flags are emitted only when set, so existing clusters keep byte-identical args and nothing rolls on operator upgrade; a conflicting flag inetcdOptionsstill wins, following the documented precedence. A CEL rule rejectsautoCompactionModewithoutautoCompactionRetention.NOSPACE alarm surfacing. Without a quota story the operator is silent at the worst moment: when etcd hits its backend quota it raises a NOSPACE alarm and the cluster goes read-only, while the status only shows the generic
UnhealthyMembers. The reconciler now pollsAlarmListafter the member health check (even when the health check itself errored — an alarmed member reports unhealthy, and the deferred status update still runs). An active NOSPACE alarm sets theDegradedcondition with distinct reasonDatabaseQuotaExceededand emits a Warning event per affected member with the operational runbook (compact, defragment,etcdctl alarm disarm; a raisedquotaBackendBytesonly lands on the next StatefulSet rollout). Detection only — no defrag or disarm automation.Files
api/v1alpha1/etcdcluster_types.go— three fields + CEL cross-field rule.internal/controller/utils.go—defaultArgs/createArgstake theEtcdClusterand render the flags conditionally.internal/etcdutils/etcdutils.go—AlarmList+MemberAlarm(carries the alarm type string, so CORRUPT is one branch away for a follow-up), mirroring the existing client boilerplate.internal/controller/etcdcluster_controller.go— alarm poll inperformHealthChecks,DatabaseQuotaExceededcondition reason inupdateConditions(takes precedence overUnhealthyMembers), per-member Warning event,events.k8s.ioRBAC marker.docs/configuring-etcd-options.md— new "Database size and compaction" section, including the rollout-timing caveat.config/crd/**,config/rbac/role.yaml,docs/api-references/docs.md,zz_generated.deepcopy.go— regenerated viamake generate manifests api-docs.Review
An adversarial review pass produced 2 blocking/important findings, both resolved in follow-up commits: the
mgr.GetEventRecorderrecorder emitsevents.k8s.io/v1Events, so the core-groupeventsRBAC rule alone left theDatabaseQuotaExceededevent forbidden at runtime (blocking — dedicated rule added); and the event/docs wording implied raisingspec.quotaBackendBytestakes effect immediately, but steady-state reconciles never patch the pod template (important — reworded to "next StatefulSet rollout" in both places). Remaining notes were minor or deliberately descoped (below).Testing
TestCreatingArgs(extended table): quota rendering (8Gi→--quota-backend-bytes=8589934592), compaction mode+retention, unset fields emitting no flags,etcdOptionsconflict precedence.TestUpdateConditionsNospaceAlarm: NOSPACE sets the distinctDatabaseQuotaExceededreason; unhealthy-without-alarm keeps the generic reason.TestReportNospaceAlarms: Warning event emitted with the member name resolved from the member list; nil recorder does not panic; non-NOSPACE alarms emit nothing.TestAlarmList(internal/etcdutils, against a live embedded server): no active alarms, error on invalid endpoint.make testandmake verifypass.Descoped follow-ups
(1) No kubebuilder defaults for the new fields, deviating from the brief: CRD-defaulting quota/compaction to etcd's defaults would inject new container args into every existing StatefulSet and roll all clusters on operator upgrade; flags render only when set, and etcd's defaults (2GiB quota, 8GiB recommended max) are documented in the field comments. (2) Defrag automation, alarm disarm, and any
LastDefragTimestatus field — explicit follow-up per the brief. (3) CORRUPT alarm surfacing —etcdutils.MemberAlarmcarries the type string so a follow-up is one condition branch, but this PR reacts to NOSPACE only. (4) Numeric CEL bounds onquotaBackendBytes(e.g.<= 8Gi) —quantity()CEL comparisons are version-sensitive and etcd only warns above 8GiB; comment-documented instead. (5) No per-member status field for alarms and no new status API — condition + event only. (6) No sample-manifest or e2e additions (thetest/e2e/etcd_options_test.gopattern would need kind); unit/fake coverage only, matching #382-sized PR precedent. (7) TLS-aware alarm client — theetcdutilshelpers, including the existinghealthCheckpath, are already plaintext-only onorigin/main;AlarmListfollows the same pattern, and fixing TLS client wiring is a separate gap.🤖 Generated with Claude Code
PR series — operability fixes, TLS & EtcdMirror
Small single-purpose PRs from live kind-cluster testing of the operator. Each stands alone unless an After is listed. → = this PR.
events.k8s.ioRBAC so operator Events are actually recordedmembers[]/leaderIDfrom one health snapshot (consistent leader)altNames.ipAddressesinto certificatesvalidityDuration(365d,100d12h) as documentedDegradedcondition (was empty status)--max-concurrent-reconciles)podTemplate.specscheduling fields: topologySpread, resources, priorityClass, schedulerNameconfig/surfacespec.tls.{peer,client}surfaces (breaking alpha API)TLSReadycondition + TLS lifecycle EventsPeerCANotSharedEtcdMirrorCRD: one-way cross-cluster replication API (types + CEL)pkg/mirroragentreplication engine (fenced checkpoint-in-target)mirror-agentbinary: config/TLS-reload/statusz/metrics + kind e2e/metricsendpointEtcdBackupCR → object storage (S3/GCS)🟢 ready · ⚪ draft